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METHOD, APPARATUS/ COMPUTER PROGRAM AND STORAGE MEDIUM 
FOR RECORDING A MOVEMENT OF A USER UNIT 

Field of the invention 

The present invention generally relates to recording 
of a movement and, more specifically, to a method for 
recording a movement of a user unit over a base, which is 
5 provided with a position code. The recording of the 

movement may e.g. be carried out in order to electron- 
ically record handwritten information. 

The invention also relates to an apparatus, a 
computer program and a storage medium which can be used 
10 for recording a movement. 
Background Art 

It is known to digitize handwriting by determining 
the movement of a pen that is -used for providing the 
handwriting. The movement may e.g. be determined by one 
15 or more accelerometers mounted to the pen. Another 

alternative is to use a base with a position code and to 
provide the pen with a sensor which records the position 
code locally at the tip of the pen during the pen' s 
movement over the base. A processing unit, arranged in 
20 the pen or externally thereof, can then decode the 

position code to a series of coordinates, which reflects 
the pen's movement over the base. 

WO 01/26033, US 5,477,012, WO 00/73983 and WO 
92/17859 disclose different examples of position codes 
25 that could be used for recording handwriting. 

Moreover, WO 01/7577 9 discloses a form provided with 
a position code enabling digital recording of information 
written on the form. In addition to the position code, 
the base is here provided with a form layout, such as 
30 frames around entry fields and text indicating what 
information should be filled-in in the entry fields. 
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According to WO 01/75779, the form layout can be 
printed on the base simultaneously with the position code 
or in different runs. In order not to disturb the reading 
of the position code, different printing inks should be 
used for the form layout and the position code. The 
printing ink for the form layout should moreover be 
invisible for the sensor in the pen. If both the printing 
code and the form layout are printed with the same 
printing ink, the form layout may partly obscure the 
printing code, making it impossible to decode the 
position code in areas close to the form layout. 

However, the use of different printing inks is 
unpractical. It would be more practical if forms could be 
printed on blank papers by means of a monochrome printer, 
e.g. a monochrome laser printer. 
Summary of the invention 

One object of the invention is to make possible the 
determination of a movement of a user unit over a base 
which is provided with a position code when the position 
code is insufficiently recorded, e.g. because the 
position code is wholly or partly obscured by other 
graphics . 

Another object of the invention is to make possible 
the determination of a movement of a user unit over a 
base that is provided with a position code and other 
graphics printed with the same printing ink. 

These objects are at least partly achieved by a 
method, an apparatus, a computer program and a storage 
medium according to the independent claims. 

According to a first aspect, the invention is a 
method of recording a movement of a user unit over a 
base, which is provided with a position code, comprising 
determining an absolute position of the user unit by 
decoding the position code in at least one image in a 
5 sequence of images of the position code obtained during 
the movement of the user unit over the base; determining 
a spatial relationship between a first and a second image 
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in the sequence; and determining another absolute 
position of the user unit on the basis of the first- 
mentioned spatial relationship between the first image 
and the second image. 
5 The position code need thus not be decodeable in all 

the images in the recorded sequence in order to determine 
the movement of the user unit. If an absolute position of 
the user unit cannot be determined by decoding the posx 
tion code in a specific image, the user unit's position 
10 can as an alternative be determined based on the spatial 
relationship between two images in the sequence and a 
previously or later determined absolute position. 

When decoding the position code in order to 
determine an absolute position of the user unit, the 
15 position code in two or more images can be used if the 
position code in one of the images is insufficient for 
enabling the decoding of an absolute position. 

Thus, the images are not always processed one by 
oneto decode one absolute position from the position code 
20 in one image, but at times a position is determined based 
on information from several images. The use of several 
images for decoding a position is here referred to as a 
distributed decoding. 

The invention can be used in all situations where 
25 the position code of an image cannot be decoded for one 

or another reason. The reason for the inability to decode 
the position code need thus not be that the position code 
is partly obscured by other graphical information. The 
invention can also be used for instance when the position 
30 code is deformed or otherwise incorrect on the base so 
that decoding of an absolute position is impossible or 
when the position code is lacking locally on the base. 

If desired, the invention can of course also be used 
when the position code in every image is decodeable to an 
35 absolute image. There may e.g. be situations where the 

distributed decoding is faster and/or easier and/or more 
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secure to carry out compared to the decoding of the 
position code in every image. 

An absolute position should in this application be 
interpreted as a position defined without reference to 
other positions, e.g. a position defined by one or more 
absolute coordinates . 

The first and second images may be any pair of 
images in the sequence. The first image may be located 
before or after the second image in the sequence. The 
sequence may include only images with overlapping 
contents, or some images with overlapping contents and 
some without, or only images without overlapping 
contents. 

The position code may define positions in one or 
more dimensions. The movement that is recorded may be a 
movement in one or more dimensions. 

The determining of the spatial relationship may 
comprise determining the best overlap position between 
the first and second images. One way of doing this is to 
correlate content of at least the first and second 
images. The first and second images can be partly over- 
lapping. In this case at least part of the content of 
these images can be directly compared in order to estab- 
lish their spatial relationship. They may, but need not, 
be successive images in the sequence. 

Alternatively, the images are not overlapping, but 
they belong to a sequence of partly overlapping images so 
that their spatial relationship can be established by 
succesively determining the spatial relationship with 
intervening images in the sequence. 

As yet another alternative, the first and second 
images do not have overlapping contents but the content 
is such that the spatial relationship can nevertheless be 
determined based on the contents of the images. 
> The images can be correlated in preprocessed or 

unpreprocessed format. The images can be e.g. binarized, 
in gray scale or in colour. 
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The correlation can be made on a pixel-level or on a 
higher level, e.g. on a feature level, where specific 
features in the images are compared. Each feature can be 
represented by a number of pixels. It can have a specific 
form. Part of a form layout or other graphical informa- 
tion on the base can be used as features. Handwriting 
made by ink from a user unit can also be used for feature 
correlation . 

A position code usually consists of a plurality of 
symbols or marks, where, each symbol codes a value, e.g. a 
binary value. The spatial relationship between two images 
can therefore also be determined based on the correlation 
of symbol values or values determined for groups of 
symbols . 

15 The spatial relationship can e.g. be expressed or 

stored as a vector indicating the offset between the 
first and second images in one or two or more dimensions. 

According to a second aspect, the invention concerns 
an apparatus which comprises a processing unit, which is 
adapted to perform a method according to the first 
aspect. 

The processing unit may e.g. be realized by a suit- 
ably programmed processor, by specifically adapted hard- 
ware, such as an ASIC (Application-Specific Integrated 
25 Circuit) or an FPGA (Field Programmable Gate Array) , by 
digital or analog circuits or any combination thereof. 
The processing unit can reside in the same unit as the 
sensor that images the position code or in another unit. 
According to a third aspect the invention concerns a 
30 computer program, which comprises program code that, when 
executed in a computer, causes the computer to carry out 
a method according to the first aspect. 

The computer program can be stored and distributed 
in the form of a storage medium, such as an electronic 
35 memory like RAM, SRAM, DRAM, SDRAM, ROM, PROM, EPROM, 

EE PROM, etc; an optical disk; a magnetic disk or tape; or 
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some other commercially available storage medium. The 
storage medium can also be a propagating signal. 

Other objects, features and advantages of the inven- 
tion are apparent from the following detailed description 
of the invention , from the appended claims and from the 
drawings. 

Of course, features of the different embodiments of 
the invention may be combined in one and the same em- 
bodiment • 

Generally, all terms used in the claims are to be 
interpreted according to their ordinary meaning in the 
technical field, unless explicitly defined otherwise 
herein. All references to "a/an/ the [element, device, 
component, means, step, etc]" are to be interpreted 
openly as referring to at least one instance of said 
element, device, component, means, step, etc., unless 
explicitly stated otherwise. The steps of any method 
disclosed herein do not have to be performed in the exact 
order disclosed, unless explicitly stated otherwise. 
Brief Description of the Drawings 

The invention will now be described in more detail 
with reference to the accompanying drawings. 

Fig. 1 is a schematic view of a system in which 
distributed decoding can be used. 

Fig. 2 is a schematic illustration of how a sequence 
of images is recorded from a base with a' position code. 

Fig. 3 shows four ideal locations of a mark in a 
position code. 

Fig. 4 is a schematic illustration of a position 
code which may be used for recording a movement of a user 
unit . 

Fig. 5 is a schematic illustration of a user unit, 
in which the distributed decoding may be carried out. 

Fig. 6 is a schematic block diagram of an electronic 
circuitry part and an image sensor in the user unit shown 
in Fig. 5. 



Figs 7A and 7B show two images of a form with a form, 
layout that partially obscures the position code. 

Figs 8A and 8B show the images of Figs 7A and 7B 
after preprocessing which includes binarization. 

Figs 9A and 9B show the images of Figs 8A and 8B 
after dot localization, raster fitting and perspective 
correction . 

Figs 10A and 10B show two dot value matrices 
generated from the images of Figs 9A and 9B. 

Fig. 11 is a flow chart illustrating an example of 
an overall decoding process. 

Fig, 12 is a flowchart illustrating how the 
distributed decoding according to the invention may be 
realized. 

Fig. 13 is a diagram which shows the result of the 
correlation of the dot value matrices of Figs 10A and 
10B. 

Detailed Description of Examples of the Invention 

Fig. 1 shows a system for information management, in 
which the present invention can be used. The system com- 
prises a base 1 in the form of a sheet of paper, an appa- 
ratus for recording a movement in the form of a pen-style 
user unit 2, and an external unit 3. The sheet of paper 
constitutes a form. It is provided with a position code 
4, only a small part of which is schematically indicated 
in an enlarged scale, and a form layout 5, which here 
comprises text 6, boxes 7 to be filled-in and images 8. 
In this example, it is assumed that the position code and 
the form layout 5 are printed with the same printing ink 
so that the position code cannot be recorded in areas 
where it is obscured by the form layout. 

The user unit 2 can be used to write on the sheet of 
paper 1 in the same way as with an ordinary pen and to 
simultaneously record what is being written in digital 
form by recording of its movement. To this end, the user 
unit 2 continuously during writing records images of the 
position code at the tip of the user unit 2. Ideally, the 
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position code in each image is decoded to an absolute 
position. A sequence of such decoded positions then 
constitutes a digital representation of the movement of 
the user unit 2 over the base and, thus of what is 
written on the base 1. 

The position decoding is carried out in the user 
unit 2 and/or the external unit 3, to which the recorded 
images can be sent automatically (on-line) or when 
commanded by the user. The external unit 3 may be a PC, a 
mobile phone, a PDA (Personal Digital Assistant), a 
server on the Internet or any other unit that has the 
capacity to carry out the position decoding. For the 
digitally recorded information to correctly reflect the 
user unit's movement over the sheet of paper, it is im- 
15 portant that the position decoding be made as correctly 
and completely as possible. 

The position code 4, which will be described in more 
detail below, is made up of symbols or marks, which are 
arranged or designed according to predetermined rules. If 
a partial area of the position code is correctly imaged, 
said partial area having a predetermined minimum size, 
the position of the partial area in the total position 
code can be determined unambiguously by means of one or 
more symbols in the imaged partial area and a predeter- 
25 mined decoding algorithm. The position can be obtained 

as a pair of coordinates in a Cartesian or another coor- 
dinate system. 

The decoding functions well as long as the position 
code in the partial area is completely and correctly 
30 imaged. Under actual conditions, however, the user unit 2 
sometimes cannot image the position code entirely 
correctly, for instance because of poor lighting con- 
ditions, or because of defects in the imaging system. 
Problems can also arise if the position code is not quite 
35 accurately reproduced on the base but has been deformed 
to some extent when being applied to the base. 
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If the position code contains redundant information, 
errors affecting a limited number of symbols can under 
certain conditions be corrected. 

Furthermore, WO 03/038741, discloses a method for 
position decoding, where, in case of a decoding error, 
information determined from the partial area is compared 
or matched with information about how the position code 
is composed in a matching area close to a previously 
successfully decoded position. The best matching portion 
of the known matching area is selected, and a correspon- 
ding position is determined as the decoded position. 

If, however, too large parts of the position code is 
deformed or missing, for instance because the position 
code is partly or totally obscured by a form layout prin- 
15 ted with the same kind of printing ink as the position 
code, error correction or matching against the known 
composition of the position code will not be possible 
because too many symbols are missing or incorrect. 
Instead a distributed decoding may be used as explained 
20 in the following with reference to fig. 2. 

Fig. 2 schematically shows a small part of the base 
1 of Fig. 1 but for the sake of clarity neither the 
position code, nor the form layout is shown on the base. 
When a pen stroke is made on the base, the user unit 2 
25 simultaneously records images of the base locally at the 
tip of the user unit. Fig. 2 schematically shows an 
example of the positions in which four such images A-D 
are recorded. The images here form a sequence of part- 
ially overlapping images. Ideally, the position code of 
30 each image is decoded to a pair of coordinates. However, 
if the position code is obscured this may not be possib- 
le. Assume e.g. that no absolute position can be decoded 
from the position code in image A. Then information from 
image A is saved. If no position can be determined for 
35 image B either, the best overlap position between images 
A and B is determined. Then it is checked whether the 
position code in image A and the position code in image B 
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together can be decoded to an absolute position. Even if 
the position code in image A and image B taken alone 
cannot be decoded because it is obscured to a too large 
extent in the respective image, the combined position 
5 code from both images may be decodable to an absolute 
position. This is schematically illustrated in Fig. 2, 
with a hatched area H covering part of images A and B. If 
so, the user unit's absolute positions when recording 
image A and image B can be determined, since the spatial 
10 relationship between the two has been established. 

If an absolute position cannot be decoded from the 
combination of images A and B, the spatial relationship 
between these images is saved, e.g. as a vector V ref- 
lecting the offset in the horizontal and vertical direc- 
15 tions between images A and B. The above procedure is 

repeated until an image or combination of images is found 
in which the position code is sufficiently completely and 
correctly imaged for an absolute position to be decoded. 
Assume e.g. that an absolute position can be decoded from 
.20 the position code in image C. If the offset between 
images B and C is also determined, then the absolute 
positions of images A and B can be determined by succ- 
essively subtracting the offset from the absolute posi- 
tion of the subsequent image in the sequence. If again an 
25 absolute position cannot be decoded from the position 
code in image D, the offset between images C and D is 
determined and the absolute position of image D determ- 
ined by adding the offset to the absolute position 
decoded from the position code in image C. 
30 Before explaining the distributed decoding in 

further detail, the position code and the user unit used 
in this example will be described in somewhat more detail 
with reference to Figs 3 and 4, and Figs 5 and 6, 
respectively. A still more comprehensive description of 
35 the position code could be found in WO 03/001440. 

The position code is based on a number sequence 
which below is referred to as a difference number 



sequence. This difference number sequence has the prop- 
erty that if one takes an arbitrary partial sequence of a 
predetermined length, for instance a partial sequence 
with five difference numbers, this partial sequence 
5 always has an unambiguously determined location in the 
difference number sequence. In other words, it appears 
only once in the difference number sequence and can thus 
be used for position determination. More specifically, 
the difference number sequence "runs" along the x axis as 
10 well as along the y axis of the position code. 

The actual position code is made up of simple gra- 
phical symbols or marks 22 which can assume four diffe- 
rent values, 0-3, depending on their respective position 
in relation to a nominal position 23 or raster point. As 
15 seen in Fig. 3, each symbol 22 has the shape of a dot and 
is displaced a distance 24 in one of four different di- 
rections relative to the nominal position 23. The nominal 
position 23 is at an intersection between raster lines 21 
in an invisible or virtual raster or grid. The value 
of the symbol is determined by the direction of the dis- 
placement. Each symbol value 0-3 can be converted into 
one bit which is used for coding an x coordinate and one 
bit which is used for coding a y coordinate, i.e. into 
the pairs of bits (0,0), (0,1), (1,0), and (1,1). Thus, 
25 the coding is made separately in the x direction and the 
y direction, but the graphical coding is made with a 
symbol which is common to the x and y bits. 

The distance 24 is suitably not less than about 1/8 
and not more than about 1/4, preferably about 1/6, of the 
30 distance between two adjacent raster lines 21. The dist- 
ance can, for example, be about 300 micrometers or about 
254 micrometers. The latter distance is particularly 
suitable for printers and scanners, which often have a 
resolution that is a multiple of 100 dpi (dots per inch) . 
35 All the symbols of the position code are usually printed 
with one and the same printing ink so that the position 
code is monochrome. 
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Each position is coded by 6 * 6 symbols which can 
thus be converted into a 6 * 6 bit matrix for an x coor- 
dinate and a 6 * 6 bit matrix for a y coordinate for the 
position . If the x bit matrix is considered, this can be 
5 divided into six columns of six bits each. Each sequence 
of bits in a column constitutes a partial sequence in a 
63-bit-long cyclic main number sequence which has the 
property that if a partial sequence having a length of 
six bits is selected, this has an unambiguously deter- 
10 mined place in the main number sequence. The six columns 
can thus be converted into six position numbers or 
sequence values which correspond to six places in the 
main number sequence. Between these six position numbers, 
five difference numbers can be formed in pairs, which 
15 constitute a partial sequence of the difference number 

sequence and which thus have an unambiguously determined 
place in the same and, consequently, an unambiguously 
determined place along the x axis. For a certain x co- 
ordinate, the position numbers will vary depending on the 
20 y coordinate. On the other hand, the difference numbers 

will be the same independently of the y coordinate, since 
the position numbers always vary according to the main 
number sequence, which is repeated cyclically in the 
columns in the entire position-coding pattern. 
25 Correspondingly, six rows in the y bit matrix define 

six position numbers in the main number sequence. These 
six position numbers define five difference numbers, 
which constitute a partial sequence of the difference 
number sequence and which thus have an unambiguously 
30 determined place along the y axis. 

Fig. 4 shows a very small part of the position code 
with 4*4 symbols. The raster lines 21 are shown in the 
figure for illustration purposes only. As a rule, the 
raster is virtual. 
35 When the position code is used, images of different 

parts of the position code are recorded, as has been des- 
cribed above. As a rule, the images contain considerably 



more than 6*6 symbols. Thus the position code may be 
decoded based on different sets of 6 * 6 symbols if a 
larger number of symbols are visible in the image. Sur- 
rounding symbols may also be used in the decoding , inter 
5 alia, for error detection and/or correction. The position 
code is of a so called "floating type". It means that any 
arbitrary partial area comprising a predetermined number 
of symbols, here 6*6 symbols, will define a position and 
that at least some of the symbols in the arbitrary par- 
10 tial area contribute to the coding of more than one posi- 
tion. Otherwise stated, if the arbitrary partial area is 
moved a symbol distance up, down, to the left or to the 
right, a new position will be defined by the symbols in 
the arbitrary partial area thus moved. 
15 In the following, an example of how the user unit 2 

of Fig. 1 may be realized will be described with refer- 
ence to Figs 5 and 6. In this example, the user unit 2 
comprises a casing 2011 having approximately the same 
shape as a pen. In one short side of the casing there is 
20 an opening 2012. The short side is intended to abut 

against or be placed a short distance from the surface on 
which the position determination is to take place. 

The casing essentially contains an optics part, an 
electronic circuitry part, and a power supply. 
25 The optics part comprises a light-emitting diode 

2013 for illuminating the surface which is to be imaged 
and a light-sensitive area sensor 2014 , such as a CCD 
(Charge Coupled Device) or CMOS (Complementary Metal- 
Oxide Semiconductor) sensor, for recording a two-dimen- 
30 sional image. The user unit 2 may also comprise an opti- 
cal system, such as a mirror and/or lens system. The 
light-emitting diode can be an infrared light-emitting 
diode and the sensor can be sensitive to infrared light. 
Generally speaking, the image sensor 2014 may be 
35 replaced by any kind of sensor that is suitable for 

imaging the position code so that an image of the symbols 
is obtained in black and white, in grey scale or in 
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color. Such a sensor can be a solid-state single- or 
multi-chip device which is sensitive to electromagnetic 
radiation in any suitable wavelength range. For example, 
the sensor may include a CCD element, a CMOS element, or 
a CID element {Charge Injection Device) . Alternatively, 
the sensor may include a magnetic sensor array for detec- 
tion of a magnetic property of the marks. Still further, 
the sensor may be designed to form an image of any chem.- 
ical, acoustic, capacitive or inductive property of the 
marks . 

The power supply to the user unit is obtained from a 
battery 2015 which is mounted in a separate compartment 
in the casing. The power supply can also be obtained via 
a cable from an external power source (not shown) . 
15 The electronic circuitry part 2016 comprises a 

processor unit with a processor which is programmed to 
read images from the sensor and carry out position 
decoding on the basis of these images, as well as a 
working memory and a program memory. 
20 Furthermore, in the shown embodiment, the user unit 

comprises a pen point 2017, by way of which it is poss- 
ible to write ordinary pigment-based ink on a base provi- 
ded with position code in order to digitally record the 
handwriting. The pen point 2017 can be extendable and 
25 retractable, so that the user can control whether or not 
it is to be used. In certain applications, the user unit 
need not have a pen point at all . 

Suitably, the pigment-based ink is of a type that is 
transparent to infrared light, while the symbols of the 
30 position code absorb infrared light. By using a light- 
emitting diode which emits infrared light and a sensor 
which is sensitive to infrared light, the detection of 
the position code takes place without any interference 
between the writing and the position code. 
35 Moreover, the user unit may comprise buttons 2018 

by means of which the user is able to activate and 
control the user unit. As an alternative the user unit 
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may comprise a on-of f-sensor which activates the user 
unit when it is in contact with a base. The on-of f-sensor 
could e.g. be a pressure sensor sensing the pressure on 
the pen point. The user unit may also comprise a trans- 
5 ceiver 2019 for wireless transmission, e.g. using IR 

light f radio waves or ultrasound, of information to and 
from the user unit. The user unit may also comprise a • 
display 2 020 for showing positions, recorded information 
or other information to the user. 
10 The user unit can be divided into different physical 

casings. The sensor and other components that are neces- 
sary for capturing images of the position code and for 
transmitting them may be located in a first casing, while 
the processor unit and the other components that are 
15 necessary for carrying out the position decoding may be 
located in a second casing. 

Fig. 6 is a schematic block diagram of the electro- 
nic circuitry part 2016 and the image sensor 2014 in the 
user unit shown in Fig. 5. An ASIC 700 implements the 
20 electronic circuitry part 2016 and contains a CPU 720 as 
a main controller of the electronic circuitry. The CPU 
720 has access to a memory 730 through an on-chip data 
bus 7 60. The image sensor 2014 is implemented as a CMOS 
area sensor 750 f which is connected to the data bus 760 
25 of ASIC 700 and which is capable of producing 96x96-bit, 
256-level grayscale digital images at a frequency of 100 
Hz. An image preprocessor 710 on ASIC 700 will preprocess 
images received from the image sensor 750 and store the 
preprocessed images in memory 730. The CPU 720 will 
30 perform the decoding of the position code in the 

preprocessed images into positions. A matching unit 740 
will assist the CPU 720 in these tasks. 

In the following, the decoding process will be 
explained more in detail with reference to Figs 7-13. 
35 Figs 7-10 schematically and for illustration purpose 

only show how a pair of images may be processed. It 
should be understood that the images are normally 
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processed substantially in sequence as is evident from 
the flowcharts of Figs 11 and 12. 

During the writing by the user unit 2 on the base 1 
in fig l f the image sensor 2014 of user unit 2 captures 
images of the base, step 1110 in fig 11. The size of each 
captured image is at least the same as the partial area 
required for determining a position, but usually larger. 

Figs 7A and 7B show two exemplary images captured in 
succession from a base having a position code with a 
) plurality of dots and a form layout, which form layout in 
these images includes letters or parts thereof. It is 
evident that the images have overlapping contents and 
that the images are obtained as grayscale images. 

Each captured image is preprocessed, step 1120, so 
15 as to form a suitable input for a following step 1130, in 
which the dots are localized, the virtual raster fit to 
the dots and the perspective corrected. If necessary, the 
perspective-corrected raster with the dots is displaced 
so that a raster point is localized in the origin of the 
20 sensor coordinate system. 

Steps 1120 and 1130 may be performed by way of tech- 
niques that are well known in the art, such as filtering, 
binarization or thresholding, and utilization of Fourier 
transforms. The raster may for instance be determined as 
25 described in WO 01/26034, WO01/75783 or WO 03/049023. 

Figs 8A and 8B show the images of Figs 7A and 7B 
after preprocessing. The images are now in binary form. 
It is evident that the position code is missing in those 
parts of the images that correspond to the form layout in 
30 Figs 7A and 7B. The binarized images also comprise pixel 
formations that evidently do not pertain to the position 
code. 

Figs 9A and 9B show the images of Figs 7A and 7B 
after step 1130, i.e. after the dots 22 have been local- 
35 ized and the virtual raster 21 fit thereto. Also, the 

perspective has been corrected so that the raster 21 is a 
square raster. 
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It should be mentioned that the perspective need not 
necessarily be corrected or compensated for. The dot 
values can also be decoded from images with a non-correc- 
ted perspective, possibly with the aid of perspective 
5 information. However, it is believed that the result will 
be better if the perspective is corrected and the dot 
values decoded from the perspective-corrected dot matrix. 

The intersections of the raster lines constitute 
nominal positions 23 for the dots. If the position code 
10 is complete and correctly imaged, each nominal position • 
23 should have one associated dot, which is displaced in 
one of four directions from the nominal position 23 
depending on which value it codes. In Figs 9A and 9B, 
some of the nominal positions 23 lack however an associ- 
15 ated dot and other nominal positions have more than one 
associated dot. This is due to obscuring form layout, 
imperfect preprocessing and dot localization. 

In the following step 1140 a dot value is determined 
for each nominal point 23. Various methods can be used 
20 for this purpose. 

For instance, if a nominal point 23 has only one 
associated dot, it may be investigated whether it lies 
within a certain "safe" zone around any of the four ideal 
displacement positions shown in Fig. 3. Such "safe" zones 
25 may e.g. be defined by circular areas having their 

centers at respective one of the four ideal displacement 
positions and having a predefined radius. Thus, if a dot 
is found to lie within any such "safe" zone, the dot is 
held to have the value represented by the ideal displace- 
30 ment position that forms the center of the "safe" zone. 

If no dot value can be determined for a nominal 
point because e.g. there is no dot associated with the 
point or because two dots are placed equally far from the 
ideal placement, the position is marked by a 
35 Alternatively, for each nominal position the proba- 

bility that the associated dot(s) assume any of the four 
different possible dot values may be determined, e.g. as 
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described in WO 03/001441. Then, probability threshold 
values may be applied, such that if a certain dot encodes 
any of the dot values 0-3 at a probability in excess of 
0.8, where 0.8 is the threshold, then the dot is assigned 
5 that dot value, otherwise 

The determination of the dot values of the dots in 
Figs 9A and 9B results in the dot value matrices shown m 
Figs 10A and 10B, where 0-3 represent the dot values and 
«-« indicates that no dot value could be determined for 
10 the corresponding nominal position. 

Each one of the dot value matrices of Figs 10A and 
10B can also be separated into two bit matrices - one x 
matrix and one y matrix - by translating the dot values 
0-3 to bit pairs 0,0; 0,1; 1,0; and 1,1,. where the first 
15 bit is used in the x matrix for determination of an x 

coordinate and the second bit is used in the y matrix for 
determination of a y coordinate as described for instance 

in WO 01/26033. 

Finally, the actual decoding of the dot value 
matrices in Figs 10A and 10B to absolute positions is 
carried out in step 1150. One way of carrying out this 
step will be described more in detail with reference to 
the flow chart in Fig. 12. 

First, in step 1200, a New Image is retrieved for 
25 the decoding of the position information. The New Image 
is a captured image that has been processed as described 
above so that two bit matrices have been obtained. To 
simplify the description, these two matrices are however 
referred to as the New Image. In step 1205, it is deter- 
30 mined whether it is possible to decode the position code 
in the New Image to an absolute position. If the position 
code is insufficiently recorded, for instance because it 
is partly or wholly obscured on the base, the New Image 
may not include sufficient information for allowing an 
35 absolute position to be decoded. 

If the position code cannot be decoded, the spatial 
relationship or the relative position of the New Image 
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with regard to a Previous Image is determined, step 1210, 
by way of correlation. If there is no Previous Image 
because the New Image is the first image, the process 
simply proceeds to the next step. 

in the following step, step 1215, it is checked 
whether an absolute position is known, i.e. has been 
determined, for the Previous Image. If so, an absolute 
position is determined for the New Image, step 1225, 
using the relative position determined in step 1210, and 
stored as an absolute position of the New Image. If no 
absolute position is known, the relative position is just 
stored as a relative position of the New Image, step 
1220, and. the process proceeds to step 1270, where the 
New Image is set to be the Previous Image. However, if 
there is no Previous Image, there is evidently no known 
absolute position in step 1225 and no relative position 
to be stored in step 1220 and the process thus just 
proceeds to step 127 0. 

After step 1270, the process restarts in step 1200 

with a New Image. 

If, on the other hand, it is determined in step 1205 
that the position code is decodable to an absolute posi- 
tion, then the decoding is carried out for the New Image 
in step 1230. In the following step 1235, it is checked 
whether the absolute position is known for the Previous 

Image, if any. 

As is evident from above, the absolute position of 
the Previous Image can be known either from a decoding of 
the position code in the Previous Image or from a 
calculation based on the relative position. 

If the absolute position is known, the decoded 
absolute position is simply stored in step 1240 as an 
absolute position of the New Image and the process 
proceeds to step 127 0, after which it restarts in step 
> 1200. 

If, in the alternative, the absolute position is 
unknown, the New Image is correlated with the Previous 
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Image, step 1245, in the same way as in step 1210. 
Thereafter, the absolute position for all previous images 
without a stored associated absolute position can be 
calculated using the stored relative position (s), step 
5 1250. 

It appears from above that if at the beginning of a 
pen stroke, the position code is insufficiently recorded 
in a series of images, then finally when an absolute 
position is decodable, it will be possible to go back- 
10 wards and determining the absolute positions of the 
previous images . 

Once, however, an absolute position has been 
determined for one image, the absolute position of every 
image with which a spatial relationship is determinable, 
15 can also be determined directly, either by decoding of 
the position code in the image or by comparison of at 
least part of its content with that of the previous 
image . 

If an absolute position has been determined for two 
20 images and it has been impossible to determine a position 
for an image in between, the absolute positions of these 
two images can be used for estimating the position of the 
image in between, e.g. by interpolation. 

The correlation in steps 1210 and 1245 can be 
25 carried out in different ways. One way is to correlate 

dot values of the dot value matrices or binary values of 
the x and/or y bit matrices by checking in which overlap 
position the highest number of dot values/binary values 
correspond. An advantage of this alternative is that the 
30 Previous Image could be stored in a relatively compact 
manner. 

Another alternative would be to carry out the 
correlation on a still higher level. As previously 
explained, each sequence of bits in a column in a partial 
35 area of 6*6 symbols constitutes a partial sequence in a 
63-bit-long cyclic main number sequence which has the 
property that if a partial sequence having a length of 
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six bits is selected, this has an unambiguously deter- 
mined place in the main number sequence. The six columns 
can thus be converted into six position numbers or 
sequence values which correspond to six places in the 
5 main number sequence. Thus, those columns in the x matrix 
that are sufficiently complete can be decoded to sequence 
values and the correlation can be carried out based on 
the sequence values. In the corresponding way, rows in 
the y matrix can be decoded to sequence values and 
10 correlation performed using these sequence values. 
Evidently, other groups of symbols, which could be 
decoded to a group symbol value could also be used for 
correlation . 

A further alternative would be to correlate the 
15 Previous Image and the New Image in the preprocessed and 
binarised form, but before conversion to a dot value 
matrix or to binary value matrices. This comparison could 
be carried out in a well-known way by calculating a score 
based on the number of overlapping pixels in different 
20 overlap positions. Examples of methods that can be used 
for pixel-wise correlation are disclosed in e.g. 
WO 98/20446 and WO 99/36879. 

Yet another alternative would be to correlate the 
images as they appear before preprocessing, e.g. in 

25 grayscale form. 

Still another alternative would be to correlate the 
images using distinct features in the images, for 
instance the form layout in Figs 7A and 7B. 

Suitable correlation techniques for grayscale and 
30 feature correlation are well-known to the skilled person 
and include e.g. Sum of Square Errors and Sum of Absolute 
Differences . 

If the user unit comprises a pen point which outputs 
writing ink that is visible to the image sensor of the 
35 user unit, correlation can be carried out on the basis of 
handwriting that appears in the images. 
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If the Previous Image and the Next Image are corre- 
lated on the basis of the dot values, the following 
correlation formula f (u,v) could be used: 



£ J(iml(x, y),itri2(x +«»>' + v » 
JW> V >- (width-\u\)(height-\v\) 5 
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where I(a r b) is the equality function evaluating to 
1 if a-b and 0 otherwise, im 1 is the Previous Image, 
im 2 is the New Image, width is the width of the images, 
height is the height of the images, u and v are the 
elements of the translational vector which indicates the 
overlap position, and x and y are pixel coordinates. 
15 Fig. 13 is a diagram which shows the result of the 

correlation of the dot value matrices in Figs 10A and 
10B. In the diagram, u and v are the elements of the 
translational vector and z indicates a correlation 
measure. The peak in the diagram indicates that the New 
Image is translated by a vector (3,0) relative to the 
Previous Image. However, when the spatial relationship 
between the Previous Image and the New Image is deter- 
mined, the possible displacement of the perspective- 
corrected raster in order to have a raster point in the 
origin of the sensor coordinate system needs to be taken 
into consideration. 

The invention has been illustrated above by means of 
an example of a specific position code including dots 
that are displaced from nominal positions in a raster. 

However, the distributed decoding could equally well 
be applied in connection with other types of position 
codes. It could, e.g. be used for position codes which 
code each position with a single, more complex symbol, 
like the position code in OS 5,477,012, for other 
35 position codes which code each position with a plurality 
of symbols but in a non- floating manner, or for other 
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position codes of floating type, like WO 00/73983 or for 
any other position code. 

The invention has furthermore been illustrated with 
a sequence of overlapping images. As indicated above, a 
spatial relationship between two images in a sequence of 
images can sometimes be determined even though there is 
no overlapping content. Instead the spatial relationship 
may be determined on the basis of the contents of the 
images. The position code used in the example above is 
based on cyclical main number series, which are repeated 
in columns and rows. If, for instance, the user unit is 
moved in the horizontal direction over the position code 
and two non-overlapping images are captured, one after 
the other, and if the distance between the images is 
equal to or less than half the length of the cyclical 
main number series in any of the two possible directions, 
then the spatial relationship can be determined from a 
comparison of the contents of the images and from the 
knowledge about the cyclical main number series. This is 
of course just but an example. 

The decoding example described with reference to the 
flowchart of Fig. 12 can be expanded. After the corre- 
lation step 1210, a check may e.g. be carried out as to 
whether the combined position codes of the New Image and 
25 the Previous Image are decodable to an absolute position. 
If not, the process continues as shown with step 1215. If 
however an absolute position can be decoded, the absolute 
position of the New Image is determined and stored. If no 
absolute position is known for the Previous Image, its 
30 absolute position is also determined and stored. Then the 
New Image is set to the Previous Image in step 127 0. 

As has been mentioned above, an image may contain 
several sets 6f symbols enabling the determination of 
several different positions. However, the position of an 
35 image or of the user unit when the image was recorded is 
always defined in relation to a specific reference point 
in the coordinate system of the user unit sensor, e.g. 
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the upper left corner. It should thus be understood that 
when it is indicated above that an absolute position xs 
stored for an image, it is the absolute position of the 
reference point that is stored. If the absolute position 
5 of another point in the sensor coordinate system has been 
determined, the absolute position of the reference poxnt 
can easily be calculated. 

in the examples above, information from the Previous 
and the New Image is used. However, further images in the 
10 sequence can be stored and used in combination when 
carrying out the distributed decoding. 

A purpose of the distributed decoding is to obtain 
one position for each image in the image sequence. 
However, there could be images for which no absolute 
15 position can be decoded from the position code and for 
which no spatial relationship to another image can be 
established. For such images a position may instead be 
determined e.g. by interpolation or extrapolation. Else 
it could be decided not to record any position for such 
an image. There could also be other situations where xt 
is decided not to record any position for an image. 

Thus a continuous movement of a user unit over a 
base is recorded by means of a sequence of images of a 
position code on the base. For some of the images the 
25 position code is directly decoded to absolute positions 
of the user unit, for other images their spatial rela- 
tionship to a previous or subsequent image is determined 
and user unit positions determined there from. 

The distributed decoding could also be described as 
30 a method of recording a movement of a user unit over a 
base, which is provided with a position code, which 
method comprises recording of a sequence of images, each 
of which corresponds to a partial area of the position 
code which could be decoded to at least one position, and 
35 using position code information from more than one image 
for decoding a user unit position. 
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CLAIMS 



1. A method of recording a movement of a user unit 
5 over a base, which is provided with a position code, 
comprising 

determining an absolute position of the user unit by 
decoding the position code in at least one image in a 
sequence of images of the position code obtained during 
10 the movement of the user unit over the base; determining a 
spatial relationship between a first and a second image 
in the sequence; and 

determining another absolute position of the user 
unit onthe basis of the first-mentioned absolute position 
15 and the spatial relationship between the first image and 
the second image. 

2. The method as claimed in claim 1, wherein 
determining the absolute position of the user unit 
comprises decoding position code from at least two images 

20 in the sequence. 

3. The method as claimed in claim 1 or 2, wherein 
the determining of the spatial relationship comprises 
correlating content of the first and second images. 

4. The method as claimed in claim 1, 2 or 3, wherein 
25 the determining of the spatial relationship comprises 

correlating features of the first and second images. 

5'. The method as claimed in any one of claims 1-4, 
wherein the determining of the spatial relationship 
comprises correlating position code information in the 
30 first and second images. 

6. The method as claimed in claim 5, wherein the 
position code on the base comprises a plurality of 
symbols, each of which represents a symbol value, and 
wherein the determining ofthe spatial relationship 
35 comprises determining and correlating symbol values in 
the first and second images . 
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7. The method as claimed in claim 5, wherein the 
position code on the base comprises at least one group of 
symbols, which codes a group symbol value, and wherexn 
the determining of the spatial relationship comprises 
5 determing and correlating group symbol values in at least 
the first and second images. 

8 . The method as claimed in any one of the preceding 
claims, wherein the base, in addition to the position 
code, is provided with graphical information, which 

10 partly obscures the position code. 

9. The method as claimed in any one of the preceding 
claims, wherein the position code comprises a plurality 
of symbols, each of which is displaced in relation to a 
nominal position defined by an intersection of raster 

15 lines in a regular raster. 

10. The method as claimed in any one of the 
preceding claims, wherein the sequence of images 
comprises images with overlapping content. 

11 An apparatus for recording a movement of a user 
20 unit over a base, which is provided with a position code, 
comprising a processor unit having means for performing 
the method according to any one of claims 1-10, 

12. Computer program which comprises program code 
which, when executed in a computer, causes the computer 

25 to carry out a method according to any one of claims 1 

10. , 

13. A computer-readable storage medium on which is 

stored a computer program which, when executed in a com- 
puter, causes the computer to carry out a method accord- 
30 ing to any one of claims 1-10. 
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ABSTRACT 

A position code is recorded by capturing a sequence 
of images while writing' on the base with a user unit. The 
position code may be decoded to a sequence of absolute 
positions reflecting the movement of the user unit over 
the base. If the position code in any image is not 
sufficiently recorded, e.g. because it is obscured by 
other graphics on the base, the spatial relationship of 
the image to another image in the sequence is determined. 
If the position code in this other image in the image 
sequence is decodable to an absolute position, the 
absolute position of the image in which the position code 
15 is not decodable could nevertheless be determined by 

using the spatial relationship. In this way the movement 
of the user unit can be recorded even though the position 
code is obscured, which could be the case when the same 
printing ink is used for both the position code and other 
20 graphics on the base. 
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